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I.  Executive  Summary 


In  the  sequential  world  the  mapping  of  processes  to  a  computer  system  wais  not  a  prob¬ 
lem,  because  of  the  classic  Von  Newman  architecture.  However,  in  the  parallel  worldj 
the  mapping  of  processes  to  nodes  and  the  balancing  of  the  load  on  each  node  becomes 
an  issue  that  needs  to  be  addressed.  This  problem  enters  software  development  in  the 
implementation  pha.se  and  remains  an  issue  throughout  the  testing  and  system  integra¬ 
tion  phases.  This  report  describes  a  new  method  developed  to  improve  the  mapping 
on  a  hypercube  system.  This  new  hypersphere  mapping  approach  performs  an  optimal 
mapping  based  on  the  geometry  of  the  physical  system  and  the  communication  patterns 
of  the  processes  in  the  logical  software  system.  The  hypersphere  mapper  can  map  pro¬ 
cesses  to  nodes  where  the  number  of  processes  exceeds  the  number  of  nodes.  Other 
mapping  approaches  rely  on  heuristics  to  cluster  the  communication  graphs  into  sub¬ 
graphs  to  accomplish  this  mapping.  The  hypersphere  mapper  can  be  instrumental  in  the 
implementation  phase  of  software  development  by  specifying  an  optimal  mapping  that 
will  improve  the  performance  of  the  system.  This  method  also  provides  a  transparent 
programming  environment,  where  the  programmer  does  not  need  to  know  about  the  in¬ 
terconnection  network  of  the  physical  computer  system.  The  hypersphere  mapper  was 
tested  using  simulated  directed  and  undirected  random  communication  graphs.  These 
tests  illustrated  improvements  in  average  communication  distance  between  messages  and 
maximum  utilization  of  physical  node  links  with  some  tradeoffs  in  load  balancing.  The 
future  of  the  tool  and  integration  into  software  engineering  tools  under  development  is 
discussed. 


II.  Introduction 


A.  Motivation 

The  development  of  software  for  the  sequential  world  has  been  studied  extensively.  This 
h^ls  been  demonstrated  by  the  evolution  of  software  lifecycle  models  that  define  the  phases 
and  steps  of  software  development.  This  process  h2is  not  been  specified  for  parallel  soft¬ 
ware  development,  but  is  loosely  defined  to  include  the  phases  of  sequential  development. 
The  steps  a  parallel  programmer  must  execute  for  each  phase  may  differ  tremendously, 
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because  of  the  physical  system.  Since  sequential  software  wtis  based  on  the  von  Neu¬ 
man  architecture,  the  hardware  platform  was  stable  and  not  a  great  concern.  However, 
parallel  architectures  vary  dramatically  and  continue  to  change.  Parallel  architectures 
have  been  divided  into  two  major  classes,  SIMD(Single-Instruction  Multiple-Data)  or 
MIMD  (Multiple-Instruction  Multiple  Data)  and  are  further  divided  into  subclasses  with 
different  models  of  computation.  Millions  of  dollars  have  been  poured  into  research  to 
create  faster  parallel  machines.  In  the  near  future  some  of  these  machines  will  be  capable 
of  peak  teraflops  performance  (10*^  floating  point  operations  per  second)  [Bel92].  With 
parallel  architectures  continuing  to  change,  the  limiting  factor  is  not  the  speed.  The 
true  limiting  factor  is  developing  software  that  can  achieve  these  speeds  and  exploit  the 
power,  and  concurrency  of  massively  parallel  computers.  In  order  to  develop  this  soft¬ 
ware,  techniques  and  tools  must  be  devised  to  address  the  problems  that  arise  in  parallel 
software  development  and  to  support  the  construction  of  a  software  lifecycle  model  that 
is  tailored  for  parallel  machines.  Fig.  1  illustrates  such  a  development  lifecycle  and  will 
be  referenced  in  the  context  of  this  report.  This  lifecycle  is  meant  to  highlight  the  software 
problems  encountered  during  development  of  parallel  software.  At  this  point  in  time,  it  is 
not  all  inclusive  to  the  number  of  problems  cissociated  with  developing  parallel  software, 
but  gives  the  reader  the  idea  of  the  magnitude  of  the  effort  involved. 

Parallel  programmers  face  all  the  inherent  problems  of  sequential  development,  plus 
others  that  were  not  applicable  to  the  uniprocessor  machine.  As  mentioned  in  the  Exec¬ 
utive  Summary,  the  mapping  or  allocation  of  processes  to  nodes  and  the  load  balancing 
on  each  node  is  just  one  of  the  problems  parallel  programmers  need  to  address.  Some 
additional  problems  include,  but  are  not  limited  to,  decomposition  of  the  problem  into 
coherent  processes,  communication  between  processes,  and  synchronization  of  these  pro¬ 
cesses.  Some  of  these  problems  depend  on  the  physical  architecture  and  must  be  a  concern 
for  parallel  programmers,  whereas  it  was  not  a  concern  for  sequential  programmers.  With 


this  in  mind,  it  would  appear  that  it  is  necessary  to  know  the  physical  architecture  at  the 
beginning  of  software  development.  This  makes  the  development  of  transparent  pau-allel 
software  that  can  achieve  a  high  degree  portability  across  multiple  parallel  platforms 
more  difficult.  The  development  of  techniques  and  tools  that  can  make  parallel  software 
transparent  and  portable  is  therfore  an  important  area  of  research. 

A  new  technique  for  addressing  the  mapping  and  allocation  problem  in  distributed 
memory  hypercube  topology  systems  (commonly  called  the  hypercube  embedding  prob¬ 
lem)  while  also  considering  load  balancing  is  discussed  in  this  report.  This  hypersphere 
mapping  method  can  be  important  in  the  implementation  phase  of  software  development 
by  specifying  an  optimal  mapping  that  will  improve  the  performance  of  the  system.  This 
report  discusses  the  implementation  of  software  on  hypercube  systems,  which  have  been 
a  popular  choice  for  interconnecting  large  numbers  of  processing  elements  in  parallel 
processing  systems.  The  hypersphere  mapper  also  provides  a  transparent  programming 
environment,  because  the  programmer  does  not  have  to  know  the  underlying  intercon¬ 
nection  network  of  the  physical  architecture. 

Effectively  mapping  a  collection  of  processes  onto  the  nodes  of  a  massively  parallel 
computer  is  especially  important  when  considering  problem  domains  where  the  interpro¬ 
cess  communication  pattern  is  inherently  irregular  and/or  data  dependent.  For  example, 
in  the  area  of  computational  fluid  dynamics,  some  applications  require  that  the  solu¬ 
tion  of  a  partial  diflferential  equation  be  approximated  over  an  irregular  grid  of  discrete 
points  [Fox88].  In  other  applications,  where  the  processes  are  functionally  independent, 
the  associated  interprocess  communication  pattern  may  also  be  irregular  and  possibly 
data  dependent.  For  example,  consider  a  large  embedded  information  processing  system 
(e.g.,  a  data  fusion  system)  in  which  the  input  data  comes  from  a  collection  of  distributed 
sensors.  The  processing  of  the  sensor  data  could  be  done  using  functional  parallelism, 
i.e.,  one  process  (or  perhaps  a  collection  of  subprocesses)  may  be  performing  FFTs,  while 
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other  processes  are  involved  in  solving  systems  of  linear  equations,  sorting  integers,  etc. 
In  such  a  system,  it  is  possible  (perhaps  likely)  that  the  communication  patterns  be¬ 
tween  the  functionally  independent  processes  will  be  irregular  and  depend  on  the  values 
of  the  input  data  supplied  by  the  sensors.  Therefore,  a  mapping  technique  for  allocating 
tasks  to  processors  can  be  very  beneficial,  especially  if  it  can  do  the  work  for  the  parallel 
programmer. 

The  hypercube  embedding  problem  involves  mapping  a  collection  of  software  tasks, 
called  processes,  onto  the  nodes  of  a  hypercube  multiprocessor,  so  that  the  available 
communication  resources  are  effectively  utilized.  The  objective  is  to  determine  a  mapping 
that  minimizes  the  average  Hamming  distance  between  those  pairs  of  processes  that 
require  interprocess  communication.  This  objective  is  certainly  desirable  for  the  case 
where  routing  is  handled  in  a  store-and-forward  message  passing  f^lshion  because  the  total 
delay  in  sending  a  message  from  source  to  destination  is  proportional  to  the  number  of 
links  traversed.  For  the  case  of  circuit-switched  and  virtual  cut-through  routing  schemes, 
minimizing  the  average  distance  between  communicating  process  pairs  can  reduce  the 
total  number  of  communication  links  needed  to  establish  the  required  connections  and 
therefore  potentially  reduce  the  latency  caused  by  contention  for  a  limited  number  of 
communication  resources. 

The  hypercube  embedding  problem  is  known  to  be  NP-hard  and  several  combinatoric 
heuristics  have  been  proposed  and  evaluated  in  the  literature.  In  this  report,  a  nonlinear 
programming  approach  is  introduced  'or  solving  the  hypercube  embedding  problem.  The 
basic  idea  of  the  proposed  approach  is  to  approximate  the  discrete  space  of  a  n-  dimen¬ 
sional  hypercube,  i.e.,  {z  :  z  €  {0,1}"},  with  the  continuous  space  of  a  n-dimensional 
hypersphere,  i.e.,  {x  :  x  €  3?"  &  IkiP  =  !}•  The  mapping  problem  is  initially  solved 
in  the  continuous  domain  by  employing  the  gradient  projection  technique  to  a  contin¬ 
uously  differentiable  objective  function,  where  the  primary  objective  is  to  minimize  the 
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average  squared  Euclidean  distance  between  communicating  processes.  The  optimal  pro¬ 
cess  “locations”  from  the  solution  of  the  continuous  hypersphere  mapping  problem  are 
then  discretized  onto  the  n-dimensional  hypercube.  The  proposed  approach  can  solve, 
directly,  the  problem  of  mapping  P  processes  onto  N  nodes  for  the  general  case  where 
P  >  N.  In  contrast,  competing  embedding  heuristics  from  the  literature  can  produce 
only  one-to-one  mappings  and  therefore  cannot  be  directly  applied  when  P  >  N. 

B.  Related  Work 

The  hypercube  embedding  problem  has  been  studied  extensively  in  the  past  and  a  va¬ 
riety  of  mapping  heuristics  and  theoretical  results  have  been  reported  in  the  literature. 
[CSG89]  evaluates  twelve  hypercube  mapping  heuristics  in  terms  of  mapping  quality  and 
run  time  of  the  mapping  algorithms  themselves  for  several  classes  of  interprocess  com¬ 
munication  patterns.  In  additi<^nal  papers,  the  fundamental  theoretical  results  have  been 
established  for  the  case  where  the  communication  patterns  are  assumed  to  have  regular 
structures  such  a.s  grids  [BMS88],  trees  [Wu8.5],  binary  trees  [WagS9],  and  hypercubes 
[Bha80]. 

One  potential  drawback  of  previous  embedding  heuristics  is  that  they  produce  one-to- 
one  mappings.  In  practical  applications,  the  requirement  may  arise  to  map  P  processes 
onto  N  nodes,  where  P  >  N;  thus,  many-to-one  mappings  are  generally  required.  It  ap¬ 
pears  that  all  the  known  mapping  heuristics  (including  the  twelve  evaluated  in  [CSG89]) 
assume  P  <  N.  A  straightforward  way  to  overcome  the  one-to-one  limitation  of  these 
mapping  heuristics  is  to  initially  cluster  the  P  processes  into  N  (or  fewer)  clusters  and 
then  execute  the  mapping  heuristic  based  on  the  intercluster  communication  pattern. 
Since  the  clustering  problem  is  generally  NP-complete,  an  additional  heuristic  would  be 
required  to  do  the  initial  clustering  before  actually  executing  the  mapping  heuristic.  One 
of  the  advantages  of  the  hypersphere  mapping  algorithm  is  that  the  case  of  P  >  iV  is 
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handled  directly,  because  the  produced  mapping  solution  is  not  limited  to  the  class  of 
one-to-one  functions. 

C.  Organization  of  the  Paper 

In  Section  II,  the  general  hypercube  embedding  problem  is  formulated  in  mathematical 
terms.  A  brief  overview  of  the  proposed  algorithm,  called  hypersphere  mapper,  is  given 
in  Section  III.  The  hypersphere  mapper  utilizes  an  iterative  gradient  descent  technique 
known  as  the  gradient  projection  method,  so  a  general  overview  of  the  gradient  projection 
method  is  provided  in  Section  IV.  Section  V  contains  the  detailed  description  of  hyper¬ 
sphere  mapper.  In  Section  VI,  extensive  simulation  studies  for  hypersphere  mapper  are 
conducted  for  the  case  of  randomly  generated  process  communication  patterns.  Section 
VII  presents  the  conclusions  and  future  work. 

III.  The  Hyperchbe  Embedding  Problem 

Mapping  a  set  of  P_  processes,  denoted  V  =  {0, 1, . . . ,  F  —  1},  onto  an  =  2”  node  (i.e., 
n-dimensional)  hypercube  is  viewed  as  a  problem  of  determining  a  collection  of  P  binary 
n- vectors,  Zi  €  {z  :  z  €  {0, 1}”},  i  €  "P,  where  the  components  of  Zj  comprise  the  binary 
address  of  the  node  that  process  i  is  to  be  mapped  to.  For  instance,  for  the  case  n  =  4, 
2^0  =  [  1  1  0  0  indicates  that  process  0  is  to  be  mapped  onto  node  3. 

Let  d//(zi,  Zj)  denote  the  Hamming  distance  between  nodes  with  addresses  z,-  and  Zj, 
where  the  Hamming  distance  equals  the  number  of  differing  binary  components  associated 
with  the  vectors  z,-  and  zj.  For  example,  d//([  1  0  1  0  ]^,  [  1  1  0  0  ]^)  =  2.  Let 
W  denote  the  set  of  pairs  of  processes  that  require  communication.  Thus,  W  =  : 

€  P  &  process  i  communicates  with  process  j). 

Therefore,  for  a  given  set  W  (i.e.,  a  given  communication  pattern),  the  hypercube 
embedding  problem  involves  finding  P  binary  n-vectors,  denoted  by  zq,  Zi, . . . ,  zp_i,  that 
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minimize 

^  ^h{zi,Z})-  (//.I) 

'  '  (‘J^w 

The  standard  hypercube  embedding  problem  assumes  that  P  <  N  and  that  the  mapping 
is  a  one-to-one- function,  i.e.,  Zi  Zj,  for  all  i^j  €  V,  with  i  ^  j.  [CKV87]  has  been 
proven  that  the  standard  embedding  problem  is  NP-hard. 

IV.  Overview  of  Hypersphere  Mapper 

The  main  premise  of  the  hypersphere  mapper  approach  is  to  approximate  the  discrete 
space  of  the  hypercube,  i.e.,  {z  :  z  €  {0, 1}”},  with  a  continuous  n-dimensional  (unit  ra¬ 
dius)  hypersphere,  i.e.,  :  i  €  31?”  &:  =  1}-  The  mapping  problem  in  the  continuous 

domain  is  to  determine  a  collection  of  P  real  n-vectors,  x,  €  {i  :  x  €  3?”  &:  ||x|p  =  1}, 
i  €  V.  The  advantage  of  the  continuous  domain  is  that  techniques  from  nonlinear  pro¬ 
gramming  can  be  employed  to  solve  a  constrained  optimization  problem.  In  particular, 
by  defining  a  continuously  differentiable  objective  function,  the  values  of  x,-  are  updated 
in  an  iterative  fashion  by  using  the  gradient  projection  technique.  Geometrically,  the 
vectors  Xi  are  points  that  reside  on  the  surface  of  a  hypersphere  in  the  continuous  do¬ 
main  of  3?”.  The  application  of  the  iterative  gradient  projection  technique  acts  to  migrate 
these  points  around  the  surface  of  the  hypersphere  so  as  to  minimize  a  desired  objective 
function. 

The  objective  function  used  by  hypersphere  mapper  comprises  two  components:  the 
first  component  is  the  average  squared  Euclidean  distance  between  communicating  pairs 
of  processes,  the  second  component  is  the  average  of  the  inverted  squared  Euclidean 
distance  between  every  pair  of  processes.  The  first  component  acts  to  bring  pairs  of 
communicating  processes  close  together;  the  second  component  acts  to  “spread-out”  the 
process  locations  (and  prevent  any  pair  of  processes  from  residing  at  the  same  point  on 
the  continuous  hypersphere). 
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Fig.  2.  A  geometric  view  of  how  points  from  the  surface  of  a  continuous  hypersphere  are 
discretized  onto  the  nodes  of  a  discrete  hypercube. 

Once  a  solution  is  obtained  on  the  continuous  hypersphere,  the  solution  vectors, 
denoted  by  x*,  are  discretized  onto  the  n-dimensional  binary  hypercube  according  to 
the  sign  of  each  component.  For  example,  in  three  dimensions,  the  continuous  vector 
Xq  —  [  0.4000  —  0.5000  0.7681  discretizes  to  Zq  =  I  1  0  1  ]^,  which  indicates  that 
process  0  is  to  be  mapped  onto  node  5.  Fig.  2  depicts  a  geometric  interpretation  of  how 
points  on  the  surface  of  the  continuous  hypersphere  are  discretized  onto  the  underlying 
hypercube.  All  continuous  points  belonging  to  the  same  sector  are  discretized  onto  the 
corresponding  node.  For  the  three  dimensional  case  shown,  note  that  there  are  eight 
sectors  and  eight  nodes.  In  the  general  n-dimensional  case,  there  axe  2"  sectors  and  and 
2”  nodes. 
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V.  The  Gradient  Projection  Method 


The  gradient  projection  method  is  an  iterative  gradient  descent  technique  for  solving 
constrained  optimization  problems.  Each  iteration  of  the  gradient  projection  method 
comprises  two  parts:  (1)  updating  the  values  of  the  variables  by  moving  in  the  direction 
of  the  negative  gradient  of  the  objective  function  and  (2)  orthogonally  projecting  the 
values  of  the  variables  onto  the  constraint  space. 

Consider  the  following  general  constrained  optimization  problem: 

minimize  f{x) 

subject  to  g(x)  =  0.  (^^-2) 

The  first  part  of  the  gradient  projection  method  is  to  update  x  by  moving  in  the  direction 
of  the  negative  gradient  of  /.  Thus,  letting  denote  the  value  of  the  vector  x  at 
iteration  fc,  the  first  part  of  the  gradient  projection  method  is  given  by 

,,(*+1)  ^  x<‘)  -  <,v/(i)|„j.>,  (;k3) 

where  a  is  a  positive  scalar  called  the  stepsize. 

The  second  part  of  the  gradient  projection  method  is  to  orthogonally  project 
onto  the  constraint  space  g{x)  =  0.  This  projection  operation  is  denoted  by 

^  {IVA) 

The  mathematical  description  of  precisely  how  to  do  the  projection  for  general  constraint 
surfaces  shall  not  be  included  here.  The  interested  reader  is  referred  to  [BaS79].  A 
geometric  view  of  an  iteration  from  the  gradient  projection  method  being  applied  to  a 
general  constrained  optimization  problem  is  depicted  in  Fig.  3.  The  d2ished  lines  represent 
constant  value  contours  for  the  objective  function  /(x),  i.e.,  /(x)  =  Ci  and  /(x)  =  C2. 
The  value  of  x^*^  is  initially  updated  by  moving  in  the  direction  of  the  negative  gradient, 
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fix)  =  Cj 


/W  =  C, 


Fig.  3.  A  geometric  view  of  an  iteration  of  the  gradient  projection  method. 

i.e.,  is  updated  to  the  point  aV /(x)|^_^»).  Then,  the  updated  value  is  projected 

back  to  the  constraint  surface  y(x)  =  0. 

VI.  Detailed  Description  of  Hypersphere  Mapper 

The  detailed  description  of  hypersphere  mapper  is  divided  into  three  main  parts.  First, 
a  constrained  optimization  problem  is  formulated  for  the  problem  of  mapping  processes 
onto  the  surface  of  a  continuous  n-dimensional  hypersphere.  Second,  the  gradient  pro¬ 
jection  technique  is  applied  Jis  a  means  of  solving  the  proposed  constrained  optimization 
problem.  Finally,  the  method  of  discretizing  the  continuous  process  “locations”  (from 
the  solution  of  the  constrained  optimization  problem)  is  described.  After  presenting  the 
detailed  description  of  hypersphere  mapper,  an  illustrative  example  application  is  given. 
The  final  subsection  describes  a  simple  technique  for  incrementally  spreading-out  the 
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hypersphere  mapper  solution  so  as  to  obtain  more  uniform  mappings. 


A.  Formulating  the  Constrained  Optimization  Problem 

In  the  domain  of  the  continuous  hypersphere,  the  underlying  objective  is  to  minimize  the 
average  squared  Euclidean  distance  between  pairs  of  communicating  processes,  where  the 
processes  are  located  on  the  surfiice  of  an  n-dimensionaJ  hypersphere  in  Si".  Thus,  given 
W  (i.e.,  the  set  of  pairs  of  processes  that  require  communication),  the  problem  is  to  find 
P  real  n-vectors,  i,  G  {i :  i  G  S"  fe  =  1},  i  g  "P,  that  minimize 


|W1  ^ 


(Kl) 


where  Xj  =  (  x..o  x.,i  •  •  •  x,,„_i)^  and  jjxj  -  Xjll^  = 

It  is  also  desirable  to  enforce  that  x,  ^  xj,  for  all  i,j  6  V,  with  i  j.  To  enforce  this 
constraint,  the  following  penalty  term  is  proposed: 


P-2  P-l 

E  E 


(K2) 


which  is  the  average  of  the  inverted  squared  Euclidean  distance  between  all  pairs  of 
processes.  The  desired  objective  function  comes  by  adding  Equations  V.l  and  V.2: 


The  first  term  in  Equation  y.3  acts  to  bring  the  communicating  pairs  of  processes  close 
together  while  the  second  terms  ensures  that  no  two  processes  reside  at  the  same  position. 

Combining  the  object: ■^3  ^unction  of  Equation  K3  with  the  constraint  that  the  pro¬ 
cesses  must  be  located  on  the  surface  of  an  n-dimensional  hypersphere  results  in  the 
following  constrained  optimization  problem: 

2 


mmimize  |  ^  X;l|^  +  > 


subject  to  ||x,||^  =  1,  for  all  i  €  V. 


{VA) 

iV.5) 
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B.  Application  of  the  Gradient  Projection  Method 

Note  that  the  constrained  optimization  problem  of  Equations  VA  and  V.b  can  be  ex¬ 
pressed  as: 

minimize  f{x)  (^-6) 

subject  to  <7(x)  =  0, 

where 

f(x)  = 


{w,.: 


..j)€W 


,  P-2  P-\  1  'I 


9{^)  = 


/  lkoll^-1  \ 

llxllp  - 1 

V  ||xp-,||^-l  y 


(V.9) 


and 


X  = 


f  Xo  ^ 


(KIO) 


\  / 

Derivation  of  the  Gradient 

Recall  from  Equation  /K3  that  the  first  part  of  each  iteration  of  the  gradient  projection 
method  requires  the  gradient  of  /(z),  i.e.,  V /(z).  By  differentiating  terms  from  /(z),  it 
can  be  shown  that 


^{llx.-^,lP}  =  2(x.-z,), 


{llzi  -  Zjll^}  =  2(zj  -  Zi), 
Izi  {llzi-Zjlp}  “  ||Zi-Z,||<^^^ 


1. 

dx 


and 


dxj  { ||z,  -  ZjIp}  Hz.  - 
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Define  W(i)  =  { j  :  j  €  P  fc  process  i  and  process  j  communicate  }.  Also,  define  V{i)  = 


V  —  {t}.  With  these  definitions,  the  gradient  of  /(x)  is  expressed  as 


V/(x)  = 


1^>€S(0)(^0  Xj)  + 

~  +  P(P-i)  i6?(i) 


(Kll) 


I  ®i)  +  F(P-l)  j€7»^-l)l|!rpir-»/||«  / 

Thus,  the  first  part  of  the  iteration  for  the  gradient  projection  method  (i.e.,  moving  in 
the  direction  of  the  negative  gradient)  is  given  by 


(K12) 


where  V/(x)|^_j5(i.)  is  computed  according  to  Equation  V.ll. 

Derivation  of  tne  Projection 

After  the  update  of  Equation  V.12  is  computed,  the  second  part  of  the  gradient  projection 
method  requires  that  the  resultant  x^*+*J  be  projected  back  to  the  surface  ^(x)  =  0.  This 
amounts  to  simply  normalizing  each  x,-  to  be  unit  length  (in  the  Euclidean  sense).  Thus, 
the  projection  operator  is  defined  by 


i9(x)=0  _ 


(V.13) 


ll*p-ill 


So,  the  second  part  of  the  iteration  for  the  gradient  projection  method  is  given  by 


(K14) 


where  is  computed  according  to  Equation  V.13. 

After  a  sufficient  number  of  iterations,  the  grcidient  projection  method  converges  to 
a  fixed- point,  i.e.,  the  vector  x^*')  converges  to  a  fixed  point  x*  as  k  increases. 


C.  Discretizing  onto  the  Hypercube 


The  final  part  of  hypersphere  mapper  is  to  convert  the  solution  vector  z*,  which  has 
continuous  real  components,  to  z*,  which  has  discrete  components  in  the  set  {0, 1}.  The 
conversion  is  done  by  simply  discretizing  each  component  of  z  (to  either  zero  or  one) 
according  to  their  signs.  Formally,  for  <iny  <  €  the  unit  step  function  is  defined  as 


“(0  = 


{ 


1  if  i  >  0 
0  if  <  <  0 


Therefore,  the  discretization  is  defined  by 


2*  =  «(z*), 

where  it  is  understood  that  «(•)  is  applied  to  each  component  of  z*. 


D.  An  Illustrative  Example 

The  goal  here  is  to  illustrate  how  the  process  locations  move  around  the  surface  of  the 
hypersphere  from  one  iteration  to  the  next.  Of  particular  interest  is  to  show  how  the 
relative  locations  of  the  processes  (for  a  given  communication  pattern,  W)  €iffect  the 
iterative  migration  to  the  fixed  point. 

Consider  the  problem  of  mapping  eight  processes,  V  =  {0, 1, . . . ,  7},  onto  the  nodes 
of  a  3-^imensional  hypercube.  Assume  the  communication  pattern  is  W  =  {(0,4),  (0, 7), 
(1,7),  (1,6),  (2,4),  (2, 5),  (3,5),  (3,6)}.  A  graphical  view  of  the  assumed  process  communi¬ 
cation  pattern,  i.e.,  W,  is  shown  in  Fig.  4.  The  resulting  iteration  data  from  hypersphere 
mapper  is  given  in  Table  I.  Fig.  5  shows  a  3-dimensional  perspective  of  the  position  vec¬ 
tors  for  iterations  =  0, 1 , . . . ,  5.  The  positions  are  depicted  by  the  dark  vectors  and  the 
underlying  hypersphere/hypercube  structure  is  superimposed  in  gray. 

From  Table  I,  note  that  the  initial  values  of  the  continuous  location  vectors  (i.e.,  the 
values  of  zp\  i  =  0, 1, . . . ,  7)  were  selected  so  that  process  i  is  (inititilly)  mapped  to  node 
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Table  I.  Iterations  from  Hypersphere  Mapper 


/(x‘) 
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Fig.  4.  Graphical  view  of  the  example  process  communication  pattern  W  =  {(0,4),  (0,7), 
(1,7),(1,6),(2,4),(2,5),(3,5),(3,6)}. 

i,  t  =  0, 1, ...  ,7.  After  each  iteration  k,  the  resulting  values  of  the  continuous  location 
vectors,  and  the  associated  discretized  location  vectors,  zj^\  are  given.  Also,  the 
corresponding  values  of  the  continuous  objective  function,  /(x^*^)  (refer  to  Equation 
K8),  and  the  cissociated  average  Hamming  distance,  ///(z^^^)  (refer  to  Equation  II.l), 
are  given.  The  fact  that  the  value  of  ///(x^*l)  decreases  as  k  increases  indicates  that  the 
continuous  objective  function  does  indeed  capture  the  essence  of  the  discrete  mapping 
problem. 

Note  that  the  process  to  node  mapping  produced  by  hypersphere  mapper  (taken  from 
iteration  A:  =  10  of  Table  I)  is: 

process  0  node  4 
process  1  —>■  node  7 
process  2  — ►  node  0 
process  3  — »  node  3 
process  4  —*  node  4 
process  5  — »  node  1 
process  6  — »  node  7 
process  7  — »  node  5, 


which  is  not  a  one-to-one  mapping  because  two  processes  are  mapped  to  node  4  and  two 
processes  are  mapped  to  node  7  (and  no  processes  are  mapped  to  node  2  nor  node  6). 
The  average  Hamming  distance  between  communicating  processes  for  this  mapping  is 
0.75. 

It  can  be  verified  (through  an  exhaustive  search,  for  example)  that  an  optimal  one- 
to-one  mapping  is  given  by; 

process  0  — >  node  5 
process  1  node  6 
process  2  — ♦  node  0 
process  3  — »  node  3 
process  4  — +  node  4 
process  5  — ►  node  1 
process  6  node  2 
process  7  — ►  node  7. 

The  above  optimal  one-to-one  mapping  has  an  associated  average  Hamming  distance  of 
1.00.  Thus,  for  this  example,  the  solution  from  hypersphere  mapper  produces  a  mapping 
that  is  superior,  in  terms  of  the  average  Hamming  distance,  to  the  best  possible  one-to- 
one  mapping  (i.e.,  0.75  versus  1.00)  at  the  “expense”  of  not  being  one-to-one. 


E.  Converting  Hypersphere  Mapper  Solutions  into  One-To-One  Mappings 

An  intuitive  approach  is  introduced  here  for  modifying  the  continuous  solution  produced 
by  hypersphere  mapper  so  that  after  discretization,  the  resulting  mapping  will  be  one- 
to-one  (or  to  within  a  specified  degree  of  closeness  to  being  one-to-one).  The  basic  idea 
of  the  approach  is  the  incrementally  spread  continuous  process  locations  out  of  “over- 
populated”  sectors  and  into  nearby  “under-populated”  sectors.  The  detailed  description 
of  the  approach  requires  the  following  definition. 

In  the  space  of  a  continuous  n-dimensional  hypersphere,  let  =  (  c,',o  Ci,\  •  •  •  )^ 

€  {x  :  X  €  3?"  &  ||2;|p  =  1}  denote  the  center  of  sector  z.  which  is  defined  for  all  i  = 
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Ci,j  — 


0, 1, . . . ,  2"  —  1  cis  follows: 

if  the  jth  bit  of  the  binary  representation  of  i  is  unity 
—  if  the  jth  bit  of  the  binary  representation  of  i  is  zero. 

For  instance,  for  the  3-dimensional  case,  A)  =  (~^  ^  ~ 

(  —  ^  ^  ^  center  vectors  define  the  geometric  centers  for  each  of  the 

2”  sectors  associated  with  an  n-dimensional  hypersphere. 

The  algorithm  for  incrementally  spreading  out  the  continuous  solution  from  hyper¬ 
sphere  mapper  operates  as  follows.  First,  those  sectors  having  strictly  more  than  f^] 
processes  residing  in  them  are  flagged  as  being  “over-populated”  and  those  having  no 
more  than  processes  residing  in  them  are  flagged  as  being  “under-populated.”  The 
spreading  procedure  takes  place  in  n  consecutive  phases,  denoted  by  phase  1  through 
phase  n.  During  phase  i,  those  position  vectors  residing  in  over-populated  sectors  that 
are  within  a  Euclidean  distance  of  from  the  center  of  an  under-populated  sector 
are  moved  to  that  under-populated  sector.  Immediately  after  a  position  vector  is  moved 
from  an  over-populated  sector  to  an  under-populated  sector,  the  status  of  the  zissociated 
sectors’  population  is  updated.  Because  the  diameter  of  the  continuous  hypersphere  is  2 
(in  the  Euclidean  sense),  after  phase  n  of  the  procedure,  it  is  clear  that  there  will  be  no 
over-populated  nor  under-populated  sectors.  Also,  the  mapping  associated  with  phase  j 
of  the  spreading  procedure  is  generically  a  more  uniform  mapping  than  the  one  associated 
with  phase  i,  for  all  i  <  j. 

The  spreading  procedure  weis  applied  to  the  illustrative  example  of  the  previous  sub¬ 
section  (i.e.,  the  problem  of  mapping  8  processes  onto  8  nodes  assuming  the  communica¬ 
tion  pattern  W  of  Fig.  4).  The  results  of  applying  the  spreading  procedure  are  depicted 
in  Fig.  6.  Without  application  of  the  spreading  procedure,  note  that  processes  0  and  4 
are  both  mapped  to  node  4,  processes  1  and  6  are  both  mapped  to  node  7,  and  no  pro¬ 
cesses  are  mapped  to  either  node  2  nor  6.  Phase  1  of  spreading  moves  process  4  from 
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the  over-populated  node  4  onto  the  under-populated  node  6.  Phase  2  of  spreading  moves 
process  6  from  the  over- populated  node  7  onto  the  under-populated  node  2. 


VII.  Evaluation  of  Hypersphere  Mapper 
A.  Comparison  with  Other  Hypercube  Embedding  Heuristics 

In  reference  [GSG89],  evaluations  and  comparisons  (based  on  simulation  studies)  are 
reported  for  twelve  hypercube  embedding  heuristics.  Several  classes  of  process  communi¬ 
cation  patterns  are  assumed,  including  random  patterns,  permuted  hypercube  patterns, 
and  tree  patterns.  Of  the  heuristics  evaluated  (and  for  all  of  the  assumed  process  com¬ 
munication  patterns),  the  simulated  annealing  heuristic  developed  in  [CSG89]  produced 
mappings  with  the  smallest  average  Hamming  distance.^  In  contrast,  a  greedy  algorithm 
of  [ChG88]  generally  produced  the  poorest  mappings  (excluding  the  so-called  default  or 
random  mapping  schemes,  which  map  processes  to  nodes  indepei  lont  of  the  given  pro¬ 
cess  communication  pattern).  It  is  also  reported  that  the  (typical)  running  time  for  the 
simulated  annealing  algorithm  is  around  four  orders  of  magnitude  larger  than  that  of  the 
greedy  algorithm.  Both  the  quality  of  the  mappings  and  the  running  times  associated 
with  the  other  ten  heuristics  were  shown  to  lie  within  the  bounds  defined  by  the  simulated 
annealing  and  greedy  algorithms.  For  a  detailed  description,  evaluation,  and  comparison 
of  the  twelve  heuristics,  refer  to  [CSG89]. 

In  the  present  report,  simulation  studies  are  used  to  provide  an  indication  of  how 
the  quality  of  mappings  produced  by  hypersphere  mapper  compare  to  the  quality  of 
mappings  produced  by  the  other  known  embedding  heuristics.  The  simulation  study 
carried  out  in  this  subsection  involves  mapping  128  processes  onto  the  128  nodes  of  a 
7-dimensional  hypercube.  The  assumed  communication  pattern  is  generated  by  selecting 
process  source-destination  pairs  at  random.  The  expected  number  of  randomly  selected 
^ Refer  to  [KGV83]  for  general  background  material  on  the  simulated  annealing  approach. 
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hypersphere  mapper  solution  hypersphere  mapper  solution 

with  no  spreading  with  phase  I  spreading 


hypersphere  mapper  solution  hypersphere  mapper  solution 

with  phase  2  spreading  with  phase  3  spreading 

Fig.  6.  Applying  the  spreading  procedure  to  convert  the  solution  from  hypersphere 
mapper  into  a  one-to-one  function.  The  notation  “{A,  B}”  beside  node  address  “060” 
means  that  processes  A  and  B  are  mapped  onto  the  node  labeled  abc. 
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source-destination  pairs  is  set  at  448,  i.e.,  £^(|W|]  =  448.  This  exact  same  simulation 
study  Wtis  one  of  the  experiments  used  in  evaluating  the  heuristics  in  [CSG89]. 

The  results  of  simulation  studies  for  the  hypersphere  mapper,  simulated  annealing, 
greedy,  and  default  algorithms  (averaged  over  100  runs)  are  summarized  in  Table  II.  The 
column  labeled  fniz)  denotes  the  average  Hamming  distance  between  mapped  commu¬ 
nicating  processes.  The  column  labeled  denotes  the  sample  variance  of  the  number 
of  processes  mapped  to  each  node,  which  is  defined  by 

^  E  (P"'n(0  -  I)  , 

where  pmn(i)  denotes  the  number  of  processes  mapped  to  node  i.  The  term  ^  represents 
the  average  number  of  nodes  mapped  to  each  node,  which  equals  unity  for  the  simulation 
of  this  subsection  because  there  are  /*  =  128  processes  being  mapped  onto  =  128 
nodes.  (In  general,  the  average  number  of  processes  mapped  onto  each  node  equals 
regardless  of  the  mapping). 

From  Table  II,  note  that  the  value  of  d-p^  is  nonzero  only  for  the  hypersphere  mapper 
because  the  mappings  produced  by  hypersphere  mapper  are  not  generically  one-to-one. 
On  the  other  hand,  the  competing  algorithms  always  produce  one-to-one  mappings  and 
thus  have  zero  values  for  d-p„,„  (i.e.,  exactly  one  process  is  mapped  to  each  node,  thus  the 
variance  of  the  number  of  processes  mapped  to  each  node  is  zero).  The  results  tabulated 
for  the  simulated  annealing  and  greedy  algorithms  were  taken  from  reference  [CSG89].^ 
The  last  row  of  Table  II  gives  the  results  of  the  default  mapping  algorithm,  which  simply 
maps  process  i  to  node  i,  for  all  i  =  0, 1, . . . ,  127. 

With  regard  to  average  Hamming  distance,  the  mappings  produced  by  hypersphere 
mapper  are  indeed  superior  to  those  produced  by  the  simulated  annealing  algorithm  (i.e., 

^The  greedy  algorithm  was  coded  and  simulated  using  our  randomly  generated  communication  pat¬ 
terns.  The  value  of  average  Hamming  distance  produced  by  our  simulation  of  the  greedy  algorithm  was 
within  2%  of  the  value  reported  in  [CSG89]. 
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Table  II.  Comaprison  Between  Hypersphere  Mapper  and  Other  Heuris¬ 
tics:  128  Processes  onto  128  Nodes  with  .E[1W1]  =  448 


Algorithm 

PlBl 

ms 

Hypersphere  Mapper 

1.889 

1.68 

Simulated  Annealing 

2.042 

0.00 

Greedy 

2.867 

0.00 

Default 

3.503 

0.00 

1.889  versus  2.042).  However,  the  mappings  produced  by  hypersphere  mapper  are  not 
generically  one-to-one;  thus,  the  corresponding  value  of  is  not  equal  to  zero. 

Table  III  shows  the  effect  of  applying  the  spreading  procedure  to  the  hypersphere  map¬ 
per  solution.  The  average  Hamming  distance  of  the  mapping  produced  after  spreading 
phase  1  is  slightly  better  than  that  of  the  simulated  annealing  algorithm  (i.e.,  2.020  versus 
2.042)  and  the  value  of  the  variance  of  the  number  of  processes  mapped  to  each  node, 
relative  to  the  hypersphere  mapper  solution  with  no  spreading,  is  reduced  by  around  60% 
(i.e.,  from  1.68  down  to  1.07).  It  is  also  noted  that  the  one-to-one  mapping  produced 
after  spreading  phase  7  is  significantly  better  than  the  mapping  produced  by  the  greedy 
algorithm  (i.e.,  2.587  versus  2.867). 

B.  The  Case  of  Having  More  Processes  than  Nodes 

A  primary  attribute  of  hypersphere  mapper  is  that  problems  involving  more  processes 
than  nodes  (i.e.,  P  >  N)  can  be  solved  directly.  In  contrast,  the  competing  hypercube 
embedding  heuristics  (including  the  twelve  evaluated  in  [CSG89])  require  P  <  N. 

Simulation  results  are  reported  here  for  the  case  of  mapping  256  processes  onto  64 
nodes.  The  assumed  process  communication  patterns  are  randomly  generated  (with  a 
specified  expected  number  of  source-destination  pairs).  Simulation  studies  are  conducted 
for  cases  where  the  expected  number  of  source- destination  pairs  is  128,  256,  512,  and 
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Table  III.  Application  of  the  Spreading  Procedure  to  Hypersphere  Map¬ 
per:  128  Processes  onto  128  Nodes  with  ^[|W|]  =  448 


spreading 

Mz) 

A  2 

^pmn 

none 

1.889 

1.68 

phase  1 

2.020 

1.07 

phase  2 

2.283 

0.40 

phase  3 

2.440 

0.15 

phase  4 

2.524 

0.06 

phase  5 

2.558 

0.03 

phase  6 

2.580 

0.01 

phase  7 

2.587 

0.00 

1024.  Table  IV  shows  the  results  of  the  simulation  studies  in  terms  of  average  Hamming 

A 

distance,  and  the  sample  variance  of  the  number  of  processes  mapped  onto  each 

node,  As  would  be  expected,  the  average  Hamming  distance  increases  as  the 

expected  number  of  source-destination  pairs  increases.  Also,  for  a  fixed  value  of  the 
expected  number  of  source-destination  pairs,  the  average  Hamming  distance  increases 
with  successive  applications  of  the  spreading  procedure.  Fig.  7  shows  histograms  for 
the  number  of  processes  mapped  to  each  node  for  the  case  of  256  source-destination 
pairs.  The  histogram  for  the  solution  from  hypersphere  mapper  (without  application  of 
the  spreading  procedure)  has  the  largest  sample  variance,  =  4.10.  At  the  other 
extreme,  the  histogram  eissociated  with  the  mappings  produced  after  spreading  phase  6 
hcis  the  smallest  sample  variance,  =  0.00.  The  mappings  produced  by  the  various 
spreading  phcises  provide  levels  of  tradeoff  between  the  values  of  fui^)  and  d’p^.  In 
practical  systems,  the  liif  cncy  caused  by  contention  for  communication  resources  can  be 
more  of  a  throughput  bottleneck  than  the  level  of  computational  load  balance.  Therefore, 
mappings  with  “perfect”  load  balance  (i.e.,  a  minimal  value  for  dp^)  may  not  always  be 
preferable. 
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Fig.  7.  Histograms  for  the  number  of  processes  mapped  onto  each  node  for  the  case  of 
mapping  256  processes  onto  64  nodes  with  ^[|W|]  =  256. 
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Table  IV.  Hypersphere  Mapper  Results;  256  Processes  onto  64  Nodes 


spreading 

E[\W\] 

=  128 

£||W|| 

=  256 

B[|W|| 

=  512 

£1|W|1 

=  1024 

BBBI 

pmn 

BBBI 

pmn 

■MB! 

pmn 

JBBi 

A  2 

^pmn 

none 

0.619 

3.06 

0.852 

4.10 

1.340 

4.16 

1.763 

7.73 

ph2ise  1 

0.850 

0.45 

1.000 

0.87 

1.433 

1.28 

1.850 

4.05 

phase  6 

0.973 

0.00 

1.168 

0.00 

1.598 

0.00 

2.110 

0.00 

VIII.  Conclusions 


A.  Summary 

The  hypersphere  mapper  method  described  in  this  report  addresses  the  mapping  and 
allocation  problems  that  arise  during  parallel  software  development.  This  method  is  a 
tool  that  maps  processes  to  nodes  for  the  parallel  programmer.  This  mapping  reduces 
the  contention  for  communication  resources,  in  addition  to  balancing  the  load  on  each 
processor.  This  algorithm  relieves  the  programmer  from  having  to  map  the  tasks  and 
balance  the  load.  The  algorithm  does  this  by  knowing  the  geometry  of  the  physical 
system  and  the  communication  patterns  of  the  processes.  This  means  it  is  beneficial 
in  the  testing  and  implementation  phcises  of  software  development.  It  also  provides  a 
transparent  programming  environment  to  the  parallel  programmer,  because  they  do  not 
have  to  know  the  interconnection  network  of  the  physical  architecture. 

The  hypersphere  mapper  approach  is  novel  in  that  it  utilizes  techniques  from  nonlin¬ 
ear  programming  as  a  means  of  solving  the  mapping  problem.  In  contrast,  competing 
mapping  algorithms  are  based  on  combinatoric  heuristics.  An  important  advantage  of 
hypersphere  mapper  over  existing  mapping  heuristics  is  that  the  case  of  mapping  more 
than  N  processes  onto  N  nodes  is  solved  directly.  These  competing  heuristics  are  limited 
to  problems  where  the  number  of  processes  is  no  larger  than  the  number  of  nodes. 


The  variance  of  the  number  of  processes  mapped  to  each  node  is  introduced  as  a 


means  of  quantifying  the  measure  of  “computational  load  balance”  eissociated  with  a 
particular  mapping.  The  average  distance  between  mapped  communicating  processes  is 
used  to  measure  the  communication  load  requirement  on  the  interconnection  network. 
Experiences  with  existing  hypercube  multiprocessors  have  indicated  that  latencies  caused 
by  contention  for  the  communication  resources  are  often  more  of  a  throughput  bottleneck 
than  the  uniformity  of  the  computational  load  balance.  Hypersphere  mapper  offers  a 
means  of  striking  a  practical  balance  between  these  two  competing  factors. 

B.  Ongoing  and  Future  Work 

The  hypersphere  mapper  could  also  be  used  for  2-D  mesh  systems  such  as  the  Intel 
Delta  or  Paragon  by  reformulating  the  objective  function.  In  the  2-D  case  the  mesh  of 
processors  could  be  wrapped  around  the  n-dimensional  hypershere  like  a  blanket  around 
a  globe.  The  processes  continuous  location  would  then  be  discretized  and  the  processes 
would  be  assigned  to  a  processor  in  that  region  of  the  hypersphere. 

Work  is  currently  underway  in  evaluating  a  variety  of  different  objective  functions.  By 
slightly  altering  the  proposed  objective  function  for  hypersphere  mapper,  the  resultant 
mappings  can  be  tuned  to  have  desired  characteristics.  For  instance,  the  nature  of  the 
resultant  mappings  can  be  changed  by  simply  multiplying  the  average  Euclidean  distance 
component  of  the  objective  function  (refer  to  Equation  VA)  by  a  scalar  weighting  factor, 
say  7.  In  particular,  the  mappings  produced  with  7  >  1  tend  to  have  smaller  values  of 
/^(z)  and  larger  values  of  (with  no  spreading),  than  the  mappings  produced  with 
7<1. 

For  cases  where  is  large  relative  to  |W|,  the  dominant  computational  time  associ¬ 
ated  with  each  of  the  gradient  projection  iterations  comes  in  computing  the  Component  of 
the  gradient  associated  with  the  average  inverted  Euclidean  distance  between  all  pairs  of 
processes  (refer  to  Equations  VA  and  Kll).  Instead  of  averaging  the  inverted  Euclidean 


distance  over  all  pairs  of  processes,  an  average  could  be  taken  over  a  smaller  set  of  pairs 
of  processes.  The  set  W  appears  to  be  a  natural  candidate.  Another  possibility  is  to 
average  over  a  randomly  generated  collection  of  process  pairs.  Some  preliminary  studies 
along  these  lines  are  currently  underway. 

Work  is  currently  underway  in  integrating  the  hypersphere  mapper  into  a  software  en¬ 
gineering  tool  for  high-performance  computing  systems.  The  tool,  called  the  Optimized 
Mapping  Alternate  Routing  System  (OMARS),  is  a  joint  in-house  effort  by  Rome  Lab¬ 
oratory  and  Purdue  University  to  develop  a  usable  tool  for  optimal  process  allocation 
in  massively  parallel  systems.  The  OMARS  tool  provides  a  framework  for  evaluating 
different  combinations  of  mapping  and  alternate  routing  for  process  communication  and 
would  be  used  in  the  implementation  and  testing  phases  of  the  software  development 
cycle.  It  provides  a  visual  depiction  of  the  logical  communication  of  the  processes  and 
a  graph  of  the  physical  system  illustrating  the  process  allocations  and  the  link  utiliza¬ 
tion.  OMARS  provides  the  users  with  several  mapping  algorithms  (one  of  which  is  the 
hypersphere  mapper)  and  routing  algorithms.  By  using  OMARS  the  user  can  choose  an 
optimal  mapping  b^lsed  on  the  hypersphere  mapper.  In  addition  to  the  default  routing, 
an  alternate  routing  can  be  calculated  based  on  the  mapping.  This  alternate  routing 
pattern  is  considered,  because  the  speedup  and  efficiency  gained  through  the  optimal 
mapping  of  the  hypersphere  could  be  lost  by  using  the  default  routing  found  in  the  sys¬ 
tem.  The  OMARS  tool  combines  optimal  process  allocation  with  alternate  routing  to 
increase  performance. 

The  hypersphere  mapper  algorithm  is  currently  applied  during  the  implementation 
and  testing  phases,  however,  it  could  be  applied  m  early  as  the  design  phase  of  the 
software  engineering  lifecycle.  The  hypersphere  mapper  is  based  on  the  geometry  of  the 
physical  system  and  the  communication  patterns  of  the  processes.  Since  the  communica¬ 
tion  between  functions  is  specified  during  the  design,  the  hypersphere  mapper  algorithm 
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could  be  applied  as  early  as  the  design  phase.  Thus,  performance  analysis  could  begin 
as  the  parallel  software  is  being  designed  and  developed. 
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